Neural network calibration for radiotherapy

ABSTRACT

Disclosed herein are systems and methods for identifying radiation therapy treatment data for patients. A processor accesses a neural network trained based on a first set of data generated from characteristic values of a first set of patients that received treatment at one or more first radiotherapy machines. The processor executes the neural network using a second set of data comprising characteristic values of a second set of patients receiving treatment at one or more second radiotherapy machines. The processor executes a calibration model using an output of the neural network based on the second set of data to output a calibration value. The processor executes the neural network using a set of characteristics of a first patient to output a first confidence score associated with a first treatment attribute. The processor then adjusts the first confidence score according to the calibration value to predict the first treatment attribute.

TECHNICAL FIELD

This application relates generally to calibrating an externally trainedneural network model to a data distribution of patients being treated ata set of radiotherapy machines.

BACKGROUND

Radiation therapy treatment planning (RTTP) is a complex process thatcontains specific guidelines, protocols and instructions adopted bydifferent medical professionals, such as clinicians, medical devicemanufacturers, treating physicians, and the like. Due to the extremenature of radiation emitted from radiotherapy machines, it is imperativethat all the instructions are precisely followed. Field Geometry, asused in the context of RTTP, refers to various attributes or settings ofa radiotherapy machine while a patient receives a prescribedradiotherapy dose. For instance, a prescribing physician may identify asource location (e.g., patient's organ to be treated or tumor to beeradicated) and a corresponding dosage. Moreover, other parties (e.g.,clinicians or machine manufacturers) may determine positioningattributes (e.g., angles) of the gantry and the patient on the couch toprovide optimum treatment.

Conventionally, identifying and applying guidelines to implementradiation therapy treatment are performed by the clinician/technician.For instance, selecting the most suitable field geometry for a patientis one aspect of RTTP that has been delegated to clinicians who usetheir subjective understanding and skill in conjunctions with variousexternal and internal guidelines to identify optimum field geometry foreach patient. However, this conventional method may be inefficient.

For instance, as the first step of the field geometry selection,treating physicians may identify the treatment modality (e.g., choosebetween the volumetric modulated arc therapy (VMAT) orintensity-modulated radiation therapy (IMRT)). Treating physicians maythen decide whether a coplanar or non-coplanar treatment is preferred.Treating physicians may then determine beam limiting device angles forthe treatment. In the case of IMRT, the beam delivery directions andnumber of beams are the specifically relevant variables that must bedecided, whereas for VMAT, the technician may need to choose the numberof arcs and their corresponding start and stop angles. For thesedecisions, each provider clinic and/or technician may have his or herown preferences and practices. For instance, a technician may prefer toplace the radiation isocenter directly onto the subject area (e.g.,tumor) and have a full arc of gantry motion around the subject area.Another technician may approach the same RTTP by having a few fixedfield directions and attempt to avoid other organs. Therefore, the sameRTTP may be interpreted in different ways, which has producedundesirable results.

SUMMARY

For the aforementioned reasons, there is a desire for a system that canadapt a computer model (e.g., a machine learning model) to predicttreatments for patients that are treated at individual sets ofradiotherapy machines (e.g., patients that are being treated atindividual radiotherapy clinics or patients that are treated in aparticular geographical region). Such sets of radiotherapy machines maybe used to treat different population distributions and/or be used byclinicians with different treatment outlooks. It may be difficult togenerate models that account for these differences given the lack oftraining data that is typically available from patients that are treatedat individual sets of radiotherapy machines (e.g., at individualradiotherapy clinics).

To account for these problems, it is desirable to train a machinelearning model using patient data from patients that are treated atdifferent sets of radiotherapy machines (e.g., using training data frommultiple radiotherapy clinics where more training data may beavailable). Once the machine learning model is adequately trained, aprovider (e.g., a clinic or a set of clinics) of a particular set ofradiotherapy machines that implements the systems and methods describedherein may use a processor to access and calibrate the machine learningmodel to make treatment predictions for patients being treated at theset of radiotherapy machines. Consequently, the provider may use apartially trained machine learning model to accurately predict RTTPpredictions such that calculating field geometry (or other radiationtherapy treatment attributes) may be tuned to patients being treated bythe radiotherapy machines that are managed by the provider. The providermay do so without training the model using training data generated frompatients that were treated at the provider's radiotherapy machines,which is often unavailable or too expensive to create.

In one embodiment, a method comprises accessing, by one or moreprocessors, a neural network trained based on a first set of datagenerated from characteristic values of a first set of patients thatreceived treatment at a set of one or more first radiotherapy machines;executing, by the one or more processors, the neural network using asecond set of data comprising characteristic values of a second set ofpatients receiving treatment at a set of one or more second radiotherapymachines to output a set of treatment attribute predictions, the secondset of data having corresponding labels indicating expected treatmentattribute predictions; executing, by the one or more processors, acalibration model using the set of treatment attribute predictions andlabels indicating expected treatment attribute predictions to output acalibration value; executing, by the one or more processors, the neuralnetwork using a set of characteristics of a first patient of the secondset of patients receiving treatment at the set of one or more secondradiotherapy machines to output a first confidence score associated witha first treatment attribute; and adjusting, by the one or moreprocessors, the first confidence score according to the calibrationvalue to predict the first treatment attribute.

In another embodiment, a system comprises a processor in communicationwith a radiotherapy machine, the processor configured to executeinstructions to: access a neural network trained based on a first set ofdata generated from characteristic values of a first set of patientsthat received treatment at a set of one or more first radiotherapymachines; execute the neural network using a second set of datacomprising characteristic values of a second set of patients receivingtreatment at a set of one or more second radiotherapy machines to outputa set of treatment attribute predictions, the second set of data havingcorresponding labels indicating expected treatment attributepredictions; execute a calibration model using the set of treatmentattribute predictions and labels indicating expected treatment attributepredictions to output a calibration value; execute the neural networkusing a set of characteristics of a first patient of the second set ofpatients receiving treatment at the set of one or more secondradiotherapy machines to output a first confidence score associated witha first treatment attribute; and adjust the first confidence scoreaccording to the calibration value to predict the first treatmentattribute.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting embodiments of the present disclosure are described by wayof example with reference to the accompanying figures, which areschematic and are not intended to be drawn to scale. Unless indicated asrepresenting the background art, the figures represent aspects of thedisclosure.

FIG. 1 illustrates components of a treatment attribute identificationsystem, according to an embodiment.

FIG. 2 illustrates a flow diagram of a process executed in a treatmentattribute identification system, according to an embodiment.

FIG. 3 illustrates an example calibration data set, according to anembodiment.

FIG. 4 illustrates an example flow diagram of a process executed in atreatment attribute identification system, according to an embodiment.

DETAILED DESCRIPTION

Reference will now be made to the illustrative embodiments depicted inthe drawings, and specific language will be used here to describe thesame. It will nevertheless be understood that no limitation of the scopeof the claims or this disclosure is thereby intended. Alterations andfurther modifications of the inventive features illustrated herein, andadditional applications of the principles of the subject matterillustrated herein, which would occur to one skilled in the relevant artand having possession of this disclosure, are to be considered withinthe scope of the subject matter disclosed herein. Other embodiments maybe used and/or other changes may be made without departing from thespirit or scope of the present disclosure. The illustrative embodimentsdescribed in the detailed description are not meant to be limiting ofthe subject matter presented.

Radiation therapy treatment planning (RTTP) may utilize neural networkmodels (or other machine learning models). For example, neural networksmay be used in automatic organ and tumor segmentation. Neural networksmay also provide decision support by, for example, suggesting treatmentmodalities or for selecting more detailed aspects of the treatment, suchas determining field geometry settings in external beam radiotherapy.Generally, the main outcome of neural network classifiers is a predictedlabel for a given unseen case. However, in a clinical setting, inaddition to the prediction accuracy (e.g., overall expectation ofcorrectly predicting labels), it may be important for a user to haveaccess to the confidence level of individual predictions so the user candetermine whether to use the prediction or to explore other options.Furthermore, it is important to avoid using overconfident or inaccuratepredictions to adjust how radiotherapy machines provide treatment.Therefore, it is important for the confidence level to be accurate.

Moreover, in a non-limiting example, when a neural network model istransferred from one clinical or non-clinical context to another, boththe prediction accuracy and the confidence of the predictions may beaffected. For instance, the distribution of data (population, treatmenttypes, protocols and practices, etc.) at individual clinics may bedifferent from each other, causing processors associated with theclinics to determine different treatments for patients. Consequently,neural networks may need to be trained to predict treatment plans forpatients that are treated at individual clinics or at specificradiotherapy machines. One approach to such training could be to retraina neural network model using site-specific patient data. However, giventhe amount of training data that is typically required to train a neuralnetwork, not all clinics may have the ability or resources to go throughthe development cycle to tailor their model to patients that use theclinic's radiotherapy machines.

By implementing the systems and methods described herein, a system mayresolve these training deficiencies by enabling an externally trainedneural network (e.g., a neural network model trained by anotherprocessor) to be used to predict RTTP attributes (e.g., treatmentattributes) for patients being treated at a set of radiotherapy machinesin a certain context (e.g., at a local clinic). The system may do so bycalibrating predictions made by the externally trained neural networkmodel to avoid the need to generate site-specific training data for aneural network model. The systems and methods provide for apost-processing calibration procedure to calibrate confidence scoresoutput by a neural network that was initially trained using data frompatients that were treated by radiotherapy machines at otherradiotherapy clinics. Such calibration techniques may enable the neuralnetwork to be trained using training data that is available while stillproviding accurate clinic-specific predictions.

Advantageously, by implementing the systems and methods describedherein, a system may avoid the costs and processing resources that aretypically required to generate large curated training data sets usingdata generated from data sources with a low amount of training data suchas individual clinics. Moreover, the solution may allow forcross-clinical comparisons for models' performance in terms ofreliability, may allow for comparing the confidence levels of differentmodels, and may prevent overconfident predictions.

As will be described below, a central server (referred to herein as theanalytics server) can train a neural network or other machine learningmodel using patient data from one or more radiotherapy clinics thatutilize sets of radiotherapy machines. In a non-limiting example, thecentral server may transfer, or a processor of a local clinic mayotherwise access, the trained neural network to a processor associatedwith the local clinic for calibration to the population at the clinic.Upon being calibrated, the neural network may predict treatmentattributes that the clinicians and/or radiotherapy machines at the localclinic may use for patient treatment. FIG. 1 is a non-limiting exampleof components of a system in which the analytics server operates.

FIG. 1 illustrates components of a treatment attribute identificationsystem 100. The system 100 may include an analytics server 110 a, systemdatabase 110 b, electronic data sources 120 a-d (collectively electronicdata sources 120), end-user devices 140 a-e (collectively end-userdevices 140), an administrator computing device 150, and radiotherapyclinics 160 a-n (collectively radiotherapy clinics 160). Theradiotherapy clinics 160 may be clinics at which patients may receiveradiotherapy treatment, in some cases via one or more radiotherapymachines located within the clinic. The above-mentioned components maybe connected to each other through a network 130. Examples of thenetwork 130 may include, but are not limited to, private or public LAN,WLAN, MAN, WAN, and the Internet. The network 130 may include wiredand/or wireless communications according to one or more standards and/orvia one or more transport mediums.

The communication over the network 130 may be performed in accordancewith various communication protocols such as Transmission ControlProtocol and Internet Protocol (TCP/IP), User Datagram Protocol (UDP),and IEEE communication protocols. In one example, the network 130 mayinclude wireless communications according to Bluetooth specificationsets or another standard or proprietary wireless communication protocol.In another example, the network 130 may also include communications overa cellular network, including, e.g., a GSM (Global System for MobileCommunications), CDMA (Code Division Multiple Access), EDGE (EnhancedData for Global Evolution) network.

The system 100 is not confined to the components described herein andmay include additional or other components, not shown for brevity, whichare to be considered within the scope of the embodiments describedherein.

The analytics server 110 a may generate and display an electronicplatform configured to use various computer models (including artificialintelligence and/or machine learning models) to identify and displaytreatment attributes (e.g., RTTP treatment attributes). The electronicplatform may include graphical user interfaces (GUI) displayed on eachelectronic data source 120, the end-user devices 140, and/or theadministrator computing device 150. An example of the electronicplatform generated and hosted by the analytics server 110 a may be aweb-based application or a website configured to be displayed ondifferent electronic devices, such as mobile devices, tablets, personalcomputer, and the like. In a non-limiting example, a physician operatingthe physician device 120 b may access the platform, input patientattributes or characteristics and other data, and further instruct theanalytics server 110 a to generate an optimized RTTP. The analyticsserver 110 a may utilize the methods and systems described herein togenerate a treatment attribute and display the results on the end-userdevices (e.g., the radiotherapy machine 140 d) or adjust theconfiguration of one of end-user devices 140. The analytics server 110 amay display the treatment attribute on the physician device 120 b itselfas well.

As described herein, treatment attributes may be or include anyattributes related to treating patients at a radiotherapy clinic and/orusing a radiotherapy machine. Treatment attributes may include, but arenot limited to, different treatment modalities, field geometry settingsfor external beam radiotherapy, side effect predictions, organ and/ortumor segmentation, machine therapy attributes, dosage administrationattributes (e.g., dosage amount), treatment frequency, treatment timing,etc. A system implementing the systems and methods described herein mayprovide calibrated predictions for one or more of any such treatmentattributes for clinicians and/or radiotherapy machines to implement totreat patients.

The analytics server 110 a may host a website accessible to usersoperating any of the electronic devices described herein (e.g., endusers), where the content presented via the various webpages may becontrolled based upon each particular user's role or viewingpermissions. The analytics server 110 a may be any computing devicecomprising a processor and non-transitory machine-readable storagecapable of executing the various tasks and processes described herein.Non-limiting examples of such computing devices may include workstationcomputers, laptop computers, server computers, and the like. While thesystem 100 includes a single analytics server 110 a, the analyticsserver 110 a may include any number of computing devices operating in adistributed computing environment, such as a cloud environment.

The analytics server 110 a may execute software applications configuredto display the electronic platform (e.g., host a website), which maygenerate and serve various webpages to each electronic data source 120and/or end-user devices 140. Different users may use the website to viewand/or interact with the predicted results.

The analytics server 110 a may be configured to require userauthentication based upon a set of user authorization credentials (e.g.,username, password, biometrics, cryptographic certificate, and thelike). The analytics server 110 a may access the system database 110 bconfigured to store user credentials, which the analytics server 110 amay be configured to reference in order to determine whether a set ofentered credentials (purportedly authenticating the user) match anappropriate set of credentials that identify and authenticate the user.

The analytics server 110 a may also store data associated with each useroperating one or more electronic data sources 120 and/or end-userdevices 140. The analytics server 110 a may use the data to weighinteractions while training various AI models accordingly. For instance,the analytics server 110 a may indicate that a user is a medicalprofessional whose inputs may be monitored and used to train the machinelearning or other computer models described herein.

The analytics server 110 a may generate and host webpages based upon aparticular user's role within the system 100. In such implementations,the user's role may be defined by data fields and input fields in userrecords stored in the system database 110 b. The analytics server 110 amay authenticate the user and may identify the user's role by executingan access directory protocol (e.g. LDAP). The analytics server 110 a maygenerate webpage content that is customized according to the user's roledefined by the user record in the system database 110 b.

The analytics server 110 a may receive RTTP data (e.g., patient andtreatment data) from a user or retrieve such data from a datarepository, analyze the data, and display the results on the electronicplatform. For instance, in a non-limiting example, the analytics server110 a may query and retrieve medical images from the database 120 d andcombine the medical images with RTTP data received from a physicianoperating the physician device 120 b. The analytics server 110 a maythen use various models (stored within the system database 110 b) toanalyze the retrieved data. The analytics server 110 a then displays theresults (e.g., RTTP including couch and gantry angles) via theelectronic platform on the administrator computing device, theelectronic physician device 120 b, and/or the end-user devices 140.

The electronic data sources 120 may represent various electronic datasources that contain, retrieve, and/or input data associated with RTTP(e.g., patient data and treatment data). For instance, the analyticsserver 110 a may use the clinic computer 120 a, physician device 120 b,server 120 c (associated with a physician and/or clinic), and database120 d (associated with the physician and/or the clinic) toretrieve/receive RTTP data associated with a particular patient'streatment plan.

End-user devices 140 may be any computing device comprising a processorand a non-transitory machine-readable storage medium capable ofperforming the various tasks and processes described herein.Non-limiting examples of an end-user device 140 may be a workstationcomputer, laptop computer, tablet computer, and server computer. Inoperation, various users may use end-user devices 140 to access the GUIoperationally managed by the analytics server 110 a. Specifically, theend-user devices 140 may include clinic computer 140 a, clinic database140 b, clinic server 140 c, a medical device, such as a CT scan machine,radiotherapy machine (e.g., a linear accelerator or a cobalt machine),and the like (140 d), and a clinic device 140 e.

The administrator computing device 150 may represent a computing deviceoperated by a system administrator. The administrator computing device150 may be configured to display data retrieved, treatment attributesgenerated by the analytics server 110 a (e.g., various analytic metricsand/or field geometry) where the system administrator can monitorvarious models utilized by the analytics server 110 a, electronic datasources 120, and/or end-user devices 140; review feedback; and/orfacilitate training or calibration of the neural networks that aremaintained by the analytic server 110 a.

In operation, a physician may access an application executing on thephysician device 120 b and input RTTP data (e.g., patient information,patient diagnosis, radiation therapy treatment attributes, etc.). Theanalytics server 110 a may then use a patient identifier to querypatient data (e.g., patient anatomy and/or medical images) from theelectronic data sources 120. The analytics server may then identify aclinic associated with the patient (e.g., clinic performing thetreatment) and retrieve the neural network that is associated with theclinic (e.g., the neural network that has been calibrated based on arepresentative set of patient data of the clinic) based on a clinicidentifier (e.g., an alphanumerical or numerical identifier that isassociated with the clinic). The analytics server 110 a may then utilizethe systems and methods described herein to generate anoptimized/uniform RTTP and display the results onto the physician device120 b, clinic computer 140 a, and/or the medical device 140 d (e.g., adisplay screen of the radiotherapy machine).

The analytics server 110 a may be in communication (real-time or nearreal-time) with the medical device 140 d, such that a server/computerhosting the medical device 140 d can adjust the medical device 140 dbased on the treatment attributes generated by the analytics server 110a. For instance, the radiotherapy machine may adjust the gantry andcouch based on angles and other attributes determined by the analyticsserver 110 a. The analytics server 110 a may transmit instructions tothe radiotherapy machines indicating any number or type of treatmentattributes (e.g., field geometry settings) to facilitate suchadjustments.

The analytics server 110 a may store machine learning models (e.g.,neural networks, random forest, support vector machines, etc.), that aretrained to predict treatment attributes to treat patients atradiotherapy clinics. The analytics server 110 a may train the machinelearning models using patient data of patients that are treated atradiotherapy machines 170 a-n of the radiotherapy clinics 160. Forinstance, the analytics server 110 a may receive patient data fromprocessors of the radiotherapy clinics 160 and generate one or more setsof labeled training data indicating treatment attributes that were usedto treat the patients at the respective radiotherapy clinics 160. Theanalytics server 110 a may input the set of labeled training data intothe stored machine learning models for supervised training to teach themachine learning models to predict confidence scores for treatmentattributes for patient treatment. The analytics server 110 a maycontinue to feed the training data into the machine learning modelsuntil the machine learning models are accurate to a threshold and storethe models in a database of the analytics server 110 a.

The machine learning models stored in the analytics server 110 a maycorrespond to individual radiotherapy clinics or otherwise differentsets of radiotherapy machines (e.g., radiotherapy machines that arelocated at individual radiotherapy clinics, are located in differentgeographical regions, treat specific types of diseases (e.g., differenttype of cancers), treat specific genders, etc.). For example, eachmachine learning model may be associated with an identifier indicatingthe radiotherapy clinic or set of radiotherapy machines for which it isconfigured to predict confidence scores for treatment attributes. Anoperator at a radiotherapy clinic may access an end-user device 140located at the clinic or access an account associated with the clinic.The operator may provide an input at a user interface that causes theend-user device 140 to transmit a request to access a machine learningmodel that is associated with the clinic and/or the radiotherapymachines located within the clinic. The request may include anidentifier associated with the machine learning model, the clinic,and/or the set of radiotherapy machines that the analytics server 110 amay use as a key in a look-up table to identify the machine learningmodel. The analytics server 110 a may receive the request and, in somecases, after authenticating the user, identify the machine learningmodel from the identifier. The analytics server 110 a may transmit theidentified machine learning model to the end-user device 140 or send analert indicating the end-user device is authorized to access the model.

Upon receipt or access to the machine learning model, the end-userdevice 140 may perform the systems and methods described herein tocalibrate the machine learning model to predict confidence scores and/ortreatment attributes for the population of patients that is generallytreated at the radiotherapy clinic and/or the respective set ofradiotherapy machines. For example, the end-user device 140 may generatea calibration data set that includes data that represents thecharacteristics of patients that are generally treated at the clinic orby the set of radiotherapy machines. The end-user device 140 may inputthe calibration data set into the machine learning model to obtainconfidence score outputs for a set of treatment attributes. For eachpatient, the end-user device 140 may identify the treatment attributethat is associated with the highest confidence score and generate apredicted treatment attribute data set from the identified treatmentattributes. The end-user device 140 may input the predicted treatmentattribute data set along with labels indicating the ground truth (e.g.,the correct prediction for each of the predicted treatment attributes)into a calibration module (e.g., a set of executable instructions)including a calibration model (e.g., a machine learning model or anoptimization model) that may be executed by the end-user device 140.Based on the input, the calibration model may output one or morecalibration parameters that the end-user device 140 may use to calibrateany confidence score predictions that the machine learning model makeswhen predicting treatment attributes.

Upon determining the calibration parameters, the end-user device 140 mayfeed patient data for new patients into the machine learning model anduse the calibration parameters to calibrate the output confidence scoresfor different treatment attributes. For example, the end-user device 140may input each of the confidence scores and the calibration parametersinto a function such as:

${q_{hat}(x)} = {1/\left( {1 - e^{- \frac{z{(x)}}{T}}} \right)}$ or${q_{hat}(x)} = {sig{m\left( \frac{z(x)}{T} \right)}}$

where x is the predicted class (e.g., field geometry setting or othertreatment attribute), q_(hat)(x) is the calibrated confidence score,z(x) is a logit where

${z(x)} = {\log\left( \frac{p_{hat}(x)}{1 - {p_{hat}(x)}} \right)}$

where p_(hat)(x) is the uncalibrated first confidence score, and T isthe calibration parameter determined by the calibration model. Theend-user device 140 may obtain a calibrated confidence score for each ofthe inputs and output the confidence scores and/or the correspondingtreatment attributes to a user on a user interface of the end-userdevice 140. A user (e.g., a patient, doctor, clinician, etc.) may viewthe confidence scores and determine which treatment attributes to usefor treatment.

Because the confidence scores have been calibrated (e.g., dampened orincreased) based on the calibration parameters, the end-user device 140may avoid displaying overconfident or inaccurate confidence scores thatoften occur in poorly trained machine learning models or models trainedusing a non-representative dataset. For example, a model withoutcalibration may predict a class correctly, but may predict too strong ofa confidence (e.g., 98%) when the confidence should have been muchlower. By calibrating the neural network, the confidence scores may beadjusted to make the confidence score predictions more accurate.

Additionally or alternatively, instead of receiving the machine learningmodel, the end-user device 140 may access the machine learning model bytransmitting instructions to the analytics server 110 a to calibrateand/or use the machine learning model as described herein. The end-userdevice 140 may transmit patient data of patients treated at therespective radiotherapy clinic and/or the set of radiotherapy machinesalong with one or more flags or settings to the analytics server 110 ato cause the analytics server 110 a to generate calibration parameters.The analytics server 110 a may generate a representative calibrationdata set from the patient data and generate one or more calibrationparameters based on the flags or settings received from the end-userdevice 140 using the systems and methods described herein. Aftercalibration, the analytics server 110 a may input patientcharacteristics into the machine learning model and generate a set ofcalibrated confidence scores. The analytics server 110 a may transmitthe calibrated confidence scores and the corresponding treatmentattributes back to the end-user device 140 for display.

FIG. 2 illustrates a flow diagram of a process executed in a treatmentattribute identification system, according to an embodiment. The method200 includes steps 210-250. However, other embodiments may includeadditional or alternative steps, or may omit one or more stepsaltogether. The method 200 is described as being executed by a dataprocessing system (e.g., a computer similar to the data source 120,end-user device 140, or the analytics server 110 a described in FIG. 1).However, one or more steps of method 200 may be executed by any numberof computing devices operating in the distributed computing systemdescribed in FIG. 1. For instance, one or more computing devices maylocally perform part or all of the steps described in FIG. 2 or a clouddevice may perform such steps.

At step 210, a data processing system (e.g., the analytics server 110 aor the end-user device 140) may access a neural network (or any othermachine learning model such as random forest or a support vectormachine) trained based on a first set of data generated fromcharacteristic values of a first set of patients that received treatmentat a set of one or more first radiotherapy machines. The neural networkmay be trained by the data processing system or by an external dataprocessing system such as an external computer or server (e.g., theanalytics server 110 a). The set of one or more first radiotherapymachines may be radiotherapy machines that are located acrossradiotherapy clinics, that are located in different geographical regions(e.g., different cities, counties, states, etc.), that treat patientswith different characteristics (e.g., that have different genders,weights, heights, body shapes, etc.), and/or that treat patients thathave different diseases (e.g., patients with different types ofcancers). Consequently, the set of patients may include patients with adiverse set of characteristics that can be used to train the neuralnetwork to predict treatment attributes for a wide range of people. Thesystems and methods described herein may be used to predict anytreatment attributes for patients.

The neural network may be trained using supervised, semi-supervised,and/or unsupervised training or with a reinforcement learning approach.For example, the neural network may be trained to predict field geometrysettings for a radiotherapy machine to use to treat patients. To do so,characteristic values of individual patients of the first set ofpatients may be input into the neural network with labels indicating thecorrect predictions for the patients. The neural network may outputfield geometry settings for individual patients based on theirrespective characteristics and the outputs can be compared against thelabels. Using back-propagation techniques, the neural network may updateits weights and/or parameters based on differences between the expectedoutput (e.g., the ground truth) and the actual outputs to better predictfuture unseen cases (e.g., field geometry settings for future patients).Similar techniques may be used to train neural networks to predict anytreatment attributes. A computer (e.g., the analytics server 110 a) maycontinue this process until the neural network is sufficiently trained(e.g., accurate above a predetermined threshold). The computer may storethe neural network in memory, in some cases upon determining the neuralnetwork has been sufficiently trained.

The data processing system may access the neural network via the cloudor by retrieving or receiving the neural network. For example, the dataprocessing system may transmit a password or token to a device storingthe neural network in the cloud to access the neural network. In anotherexample, the data processing system may receive or retrieve the neuralnetwork either automatically responsive to the neural network beingsufficiently trained or responsive to a GET request from the dataprocessing system. The data processing system may transmit a request forthe neural network responsive to a request at a user interface of thedata processing system by a user. Upon receiving the neural network, thedata processing system may store the neural network in memory forretrieval when the data processing system makes predictions fortreatment attributes for individual patients that receive treatment atthe set of radiotherapy machines with which the neural network isassociated.

At step 220, the data processing system may execute the neural networkusing a second set of data comprising characteristic values of a secondset of patients receiving treatment at a set of one or more secondradiotherapy machines to output a set of treatment attributepredictions, the second set of data may have corresponding labelsindicating expected treatment attribute predictions. The second set ofradiotherapy machines may be located at a specific radiotherapy clinic,located within a specific geographical area, treat specific populationsof patients (e.g., specific genders, weights, heights, etc.), treatspecific types of disease (e.g., tumors located in specific areas), orbe related in any other way. The data processing system may beconfigured to receive patient attributes for a patient being treated ata radiotherapy machine of the second set of radiotherapy machines andinput the patient attributes into the neural network to obtain treatmentattributes such as field geometry settings for treatment. The neuralnetwork may not be trained, or may only be partially trained, based onpatient data of patients that received treatment at the second set ofradiotherapy machines (e.g., in instances in which such data is notavailable or too expensive to generate during the training phase of theneural network). Thus, there may be a need to calibrate the neuralnetwork to provide accurate predictions for patients being treated atthe second set of radiotherapy machines.

The second set of data may be or include a calibration data set that isrepresentative of the population of people that are treated at the setof one or more second radiotherapy machines. For example, if the set ofone or more second radiotherapy machines generally treats a high ratioof females to males, the calibration data set may includecharacteristics of patients with the same or a similar ratio (e.g.,within a threshold). In another example, if the set of one or moresecond radiotherapy machines generally only treats patients that havebreast and lung cancer, the calibration data set may only includecharacteristics of patients with breast and lung cancer. The calibrationdata set may further include a similar ratio of patients with breastcancer to patients with lung cancer. Calibration data sets may similarlyinclude data of patients that is representative of multiplecharacteristics of the population of patients that were treated at theset of second radiotherapy machines. The calibration data set may berepresentative of populations of patients having any number ofcharacteristics.

The data processing system may execute the neural network bysequentially feeding the second set of data into the neural network forindividual patients. For each patient, the data processing system maygenerate a vector comprising values of the characteristics of thepatient (e.g., height, weight, gender, tumor size, tumor location, age,prescribed dosage, body mass index, image data of targets and organs,etc.) and input the vector into the neural network. The neural networkmay receive the vector and output confidence scores for differenttreatment attribute predictions based on the weights and parameters theneural network acquired during training.

For example, the neural network may be trained to predict whether to usestandard field geometry settings or customized field geometry settingsto treat patients. Upon receiving an input of characteristics of aparticular patient that is about to be treated at a radiotherapy machineof the one or more second radiotherapy machines, the neural network maypredict confidence scores for both the potential outputs of standardfield geometry settings and customized field geometry settings. Theconfidence scores may indicate the certainty that each potential outputis the correct output. The following vector illustrates exampleprediction outputs (e.g., labels associated with the highest predictedconfidence scores) for four different patients by such a neural network:

(y _(hat) ,p _(hat))=<1,0.96;0,0.98;1,0.86;0,0.9>

where y_(hat) is the predicted field geometry setting label where 1 isstandard field geometry setting and 0 is a customized field geometrysetting and p_(hat) is the confidence score associated with the label.

In another example, the neural network (or a different neural network)may be trained to predict whether a particular treatment has harmfulside effects. The neural network may have the potential outputs of “noharmful side effects (label=0),” “mild side effects (label=1),” and“serious side effects (label=2).” Upon receiving an input ofcharacteristics of a patient (which may include a delivered dosedistribution), the neural network may output confidence scores for eachpotential output indicating a likelihood that the patient willexperience side effects of the associated class. The following vectorillustrates example prediction outputs (e.g., labels associated with thehighest predicted confidence scores) for four different patients by sucha neural network:

(y _(hat) ,p _(hat))=<1,0.86;2,0.75;1,0.8;1,0.68>

where y_(hat) is the predicted side effect label according to the abovelabels and p_(hat) is the confidence score associated with the label(e.g., the likelihood the patient will experience the respective sideeffect).

In addition to patient characteristics, the calibration data set mayinclude labels indicating the correct treatment attribute predictionsfor individual patients (e.g., the ground truth of the calibration dataset). For example, the calibration data set may include labelsindicating the known correct treatment or known outcome for thecharacteristics that are input into the machine learning model. Suchlabels may be used to calibrate the neural network as will be describedbelow. The following vector illustrates example ground truth labels forfour different patients to input into two machine learning models (e.g.,a machine learning model configured to predict field geometry settinglabels and a machine learning model configured to predict side effectlabels):

(field geometry setting label,side effect label)=<0,0;0,2;1,1;0,1>.

At step 230, the data processing system may execute a calibration modelusing the set of treatment attribute predictions and labels indicatingexpected treatment attribute predictions to output a calibration value.The calibration model may be another machine learning model (e.g., aneural network, support vector machine, random forest, etc.) that istrained to predict calibration parameters for machine learning modelssuch as the neural network based on predicted outputs of the neuralnetwork and the corresponding labels that indicate the ground truth ofsuch outputs. The calibration model may be an optimization model (e.g.,a conjugate gradient solver) that can numerically or analyticallydetermine the calibration parameters based on the same inputs.

For example, the data processing system may input a calibration data setinto the neural network that includes characteristics for multiplepatients. For each patient, the neural network may predict confidencescores for multiple treatment attributes. The data processing system maycompare the confidence scores that are associated with the treatmentattributes and identify the treatment attribute that is associated withthe highest confidence score as the predicted treatment attribute. Thedata processing system may feed the identified treatment attributesalong with the label or ground truth treatment attribute thatcorresponds to the respective patient into the calibration model toobtain a calibration value specific to the neural network that is beingcalibrated.

The calibration model may be configured to determine calibrationparameters that minimize cross-entropy loss using models such as aneural network model or with the following equation:

${H\left( {p,q} \right)} = {- {\sum\limits_{x}{{y(x)}\log{q(x)}}}}$

where x indexes the different classes, y(x) is y_(true) for class x,

${{q(x)} = {{{1/\left( {1 - e^{- \frac{z{(x)}}{T}}} \right)}\mspace{14mu}{where}\mspace{14mu}{z(x)}} = {\log\left( \frac{p_{hat}(x)}{1 - {p_{hat}(x)}} \right)}}},$

and T is the calibration parameter. The following is an example inputfor two samples in a set with three classes (e.g., treatmentattributes):

z(x)=[[4.0,2.0,1.0],[5.0,6.0,1.0]]

labels=y _(true)=[[1.0,0.0,0.0],[0.0,1.0,0.0]]

By implementing the equation or a neural network, the data processingsystem may find optimal calibration parameters to use to calibrate theneural network.

Before identifying the treatment attribute that is associated with thehighest confidence score as the predicted treatment attribute, the dataprocessing system may compare the confidence score to a threshold. Thedata processing system may determine the treatment attribute isassociated with the correct prediction responsive to the confidencescore exceeding the threshold. If the confidence score does not exceedthe threshold, the data processing system may generate a null value toinput into the calibration model with a corresponding null label toavoid calibrating the neural network based on a prediction for which theneural network has low confidence. Thus, the data processing system maycalibrate the neural network only using predictions for which the neuralnetwork is confident, increasing the accuracy of the calibration andavoiding calibrating the model based on “guesses.”

The calibration parameters may include a vector or set of parameters.Such parameters may include any number of parameters depending on thecalibration algorithm that is used to calibrate the neural network. Forexample, in the case of temperature scaling, the parameters may onlyinclude one “temperature” value that can be used to calibrate the outputconfidence scores of the neural network. The temperature value may beused to dampen predictions from the neural network model, thusminimizing the number of overconfident predictions that are made by theneural network. Other calibrations methods may include histogram binningand isotonic regression. Any number of calibration parameters may beused to adjust the confidence scores that are output by the neuralnetwork.

In instances in which the calibration model is a neural network oranother machine learning model, the neural network may be trained usingany of a supervised, semi-supervised, or unsupervised training method.For example, the neural network may be trained using training dataincluding predicted labels and ground truth labels. The neural networkmay be trained using such training data until the neural network isaccurate above a threshold. Upon exceeding the accuracy threshold, thedata processing system may use the calibration neural network to predicta temperature (or other set of parameters depending on the chosencalibration model) for the neural network that is configured to predicttreatment attributes for patients being treated at the second set ofradiotherapy machines.

At step 240, the data processing system may execute the neural networkusing a set of characteristics of a first patient of the second set ofpatients receiving treatment at the set of one or more secondradiotherapy machines to output a first confidence score associated witha first treatment attribute. The data processing system may receivevalues of characteristics of the patient from a user (e.g., a clinician,doctor, or the patient themselves) via a user interface and generate afeature vector that includes the values. Additionally or instead, thedata processing system may retrieve values of characteristics of thepatient from storage to include in the feature vector responsive toreceiving an identifier of the patient. The data processing system mayinput the feature vector into the neural network and obtain an outputfrom the neural network including confidence scores for differenttreatment attributes (e.g., different field geometry settings).

The data processing system may receive the characteristics for thepatient based on a patient identifier that is provided at a userinterface. For example, a clinician may input the name of the firstpatient into the user interface at an end-user device and the end-userdevice may transmit the name to the data processing system. The dataprocessing system may use the patient's name to query a database thatincludes patient information and retrieve information about the patientsuch as the patient's electronic health data records. For instance, thedata processing system may query the database for data associated withthe patient's anatomy, such as physical data (e.g., height, weight,and/or body mass index) and/or other health-related data (e.g., bloodpressure or other data relevant to the patient receiving radiationtherapy treatment) and/or geometrical data. The data processing systemmay also retrieve data associated with current and/or previous medicaltreatments received by the patient (e.g., data associated with thepatient's previous surgeries).

If necessary, the data processing system may also analyze the patient'smedical data records to identify the needed patient characteristics. Forinstance, the data processing system may query a database to identifythe patient's body mass index (BMI). However, because many medicalrecords are not digitalized, the data processing system may not receivethe patient's BMI value using simple query techniques. As a result, thedata processing system may retrieve the patient's electronic health dataand may execute one or more analytical protocols (e.g., natural languageprocessing) to identify the patient's body mass index. In anotherexample, if the data processing system does not receive tumor data(e.g., end-points) the data processing system may execute various imagerecognition protocols and identify the tumor data.

The data processing system may receive additional data from one ormedical professionals. For instance, a treating oncologist may access aplatform generated/hosted by the data processing system and may add,remove, or revise data associated with a particular patient, such aspatient attributes, treatment attributes, tumor attributes, primary siteof treatment, tumor stage, end-point, whether the primary tumor has beenextended, and the like. Because tumor staging and the end levelattributes are sensitive information that affect patient treatment, thisinformation is typically inputted by the treating oncologist.

The data received by the data processing system (e.g., patient/treatmentdata) may belong to three categories: numerical, categorical, andvisual. Non-limiting examples of numerical values may include patientage, physical attributes, and other attributes that describe thepatient. Non-limiting examples of categorical values may includedifferent stages of treatment or disease associated with the patient.Visual data may include medical images representing the patient andhis/her treatment regions, such as CT scans or other scans illustratingthe patient's tumor.

Another example of a patient characteristic may include specific tumorlocations. More specifically, this data may indicate the primary tumorlocation with respect to the patient's centerline. This data may beinputted by the treating oncologist or may be analyzed using variousimage recognition or segmentation methods executed on the patient'smedical images. This information can also be predicted using the machinelearning model if it is not inputted by the treating oncologist (orotherwise received by the data processing system). Another patientattribute may indicate whether and how close the tumor is to othernon-diseased organs. For instance, a tumor to be eradicated may bemillimeters away from another organ. This information may change fieldgeometry, as other organs must be avoided.

Another example of a patient characteristic may include whether thepatient uses a prosthesis (e.g., hip or femoral head prosthesis). Thischaracteristic may result in a change in the patient's treatment (e.g.,a change in confidence score for a particular treatment because patientswith these conditions might require a special treatment).

The neural network may receive such characteristics about the firstpatient and output confidence scores for one or more treatmentattributes. The first confidence score may be the highest confidencescore of the confidence scores that were output by the neural networkand may correspond to the data processing system's treatment predictionfor the first patient (e.g., the first treatment attribute). The dataprocessing system may identify the first confidence score responsive tocomparing the predicted confidence scores and identifying the highestconfidence score as the first confidence score. The data processingsystem may determine the first treatment attribute associated with thefirst confidence score is the correct prediction responsive todetermining the first confidence score is the highest predictedconfidence score. In some instances, the first treatment attribute maybe a dosage amount or a field geometry setting such as an angleassociated with a couch or a gantry of the radiotherapy machine, a setof standard field geometry settings, one or more customized fieldgeometry settings, an arc length of the gantry, a number of arcs, apositioning of the couch or gantry, length of time to apply a prescribeddosage, etc. The first treatment attribute may be any treatmentattribute as described herein.

For example, the neural network may be configured to output confidencescores for different arc length field geometry settings for aradiotherapy machine. Each confidence score may be associated with adifferent arc length. The data processing system may input thecharacteristics for the first patient into the neural network and theneural network may output confidence scores for each arc length setting.The data processing system may identify the arc length field geometrysetting that is associated with the highest confidence score as thecorrect setting.

In another example, the neural network may predict confidence scores forfield geometry settings that indicate the location to place theisocenter of the radiotherapy machine and/or an arc length or path forthe gantry of the radiotherapy machine. As used herein, the isocenter(or the radiation isocenter) refers to the point in space whereradiation beams intersect when the gantry rotates (e.g., half or fullarcs) during the “beam-on” mode. For example, the neural network maypredict a high confidence score for a field geometry setting that placesthe isocenter in the middle of the tumor and has a full arc gantrymotion while the beam is on. Such may be the case when the patient has ahigh BMI or needs to avoid other machines that are treating the patient(e.g., a ventilator). In another example, the neural network may predicta high confidence score for a field geometry setting that has a fewfixed tube directions that are evenly distributed and that attempts toavoid certain structures/organs of the patient's body.

In another example, the neural network may predict a field geometrysetting for a patient to receive volumetric modulated arc therapy (VMAT)in one or more arcs. Different settings may be associated with differentnumbers of arcs. The neural network may make such predictions when thepatient exceeds a certain height and/or weight. In another example, theneural network may predict a high confidence score for a field geometrysetting of two partial arcs when a patient is connected to a ventilator.Technicians that operate radiotherapy machines at different radiotherapyclinics that treat different types of patients may treat similarpatients using different numbers of arcs and/or treatments based on theexperiences of the physicians and/or the treatments that are effectivein the area, emphasizing the need for the neural network to becalibrated to patients being treated by the radiotherapy machine and/orradiotherapy clinic.

The data processing system may determine a probability that an eventwill occur based on the output confidence score. For example, the neuralnetwork may be configured to output confidence scores that a giventreatment will produce serious side effects. The neural network mayinclude an output node associated with a serious side effect prediction.The data processing system may input the characteristics for the firstpatient into the neural network and the neural network may output aconfidence score for the serious side effect prediction. The confidencescore may indicate a percent risk of the first patient experiencingserious side effects for the treatment.

At step 250, the data processing system may adjust the first confidencescore according to the calibration value to predict the first treatmentattribute. The data processing system may adjust the first confidencescore in addition to or instead of any other confidence scores that theneural network predicted for the first patient. To adjust the firstconfidence score, the data processing system may input the firstconfidence score into a calibration module that comprises instructionsto receive confidence scores and the determined calibration parametersand adjust the confidence scores according to the determined calibrationparameters for calibration.

For example, using the calibration module, the data processing systemmay adjust the first confidence score using one of the followingequations:

${q_{hat}(x)} = {1/\left( {1 - e^{- \frac{z{(x)}}{T}}} \right)}$ or${q_{hat}(x)} = {sig{m\left( \frac{z(x)}{T} \right)}}$

where x is the predicted class (e.g., treatment attribute), q_(hat)(x)is the calibrated confidence score, z(x) is a logit where

${z(x)} = {\log\left( \frac{p_{hat}(x)}{1 - {p_{hat}(x)}} \right)}$

where p_(hat)(x) is the uncalibrated first confidence score, and T isthe calibration parameter determined by the calibration model. The dataprocessing may input the first confidence score (and any otherconfidence scores that were output by the neural network) into thecalibration module and obtain a calibrated confidence score in return.Note that the calibration module may be or include any equation orequations that use calibration parameters to calibrate prediction scoresfor treatment attributes.

Responsive to obtaining the calibrated confidence scores, the dataprocessing system may render the scores along with the correspondingtreatment attributes on a user interface. A user, clinician, or doctormay view the results and decide the best course of treatment accordingly(e.g., implement the treatment that is associated with the highestconfidence score or select a treatment from a group of treatmentsassociated with the highest confidence scores). Advantageously, becausethe data processing system may determine and render confidence scoresthat are calibrated to provide treatment recommendations to populationsof patients that are treated at the radiotherapy machine of a set ofradiotherapy machines that treat a common subset of people, patients anddoctors can view the predicted confidence scores and be confident in thescores' accuracy. Thus the patients and doctors may determine the bestcourse of action using accurate data. Systems that do not utilize suchcalibration techniques may offer overconfident predictions that doctorsand patients may not trust, causing the patients or doctors to ignorethe predictions to instead determine a new course of action.

The data processing system may only display confidence scores thatsatisfy a predetermined criteria. For example, the data processingsystem may only display the highest calibrated confidence score or apredetermined number of the highest confidence scores to a user. Thedata processing system may identify the highest calibrated confidencescores (e.g., the calibrated first confidence score) and only displaythe highest calibrated confidence scores on a user interface of adisplay. In another example, the data processing system may only displaythe highest calibrated confidence score responsive to determining theconfidence score exceeds a threshold (e.g., a predetermined thresholdset by a user). By using such criteria, the data processing system cancontrol the confidence scores and the corresponding treatment attributesthat are displayed to users, thus ensuring users only view confidencescores for treatment attributes for which the neural network madeconfident predictions and do not get distracted by other predictions.

However, responsive to determining none the output confidence scores bythe neural network satisfy the criteria (e.g., the highest confidencescore of a set of predicted outputs is lower than a threshold), the dataprocessing system may generate an alert indicating a treatment attributecould not be predicted. The data processing system may transmit thealert to be displayed on a user interface of the radiotherapy machine, aclinic computer, and/or end-user computer. The alert may include textindicating the criteria was not satisfied, the predicted confidencescores (e.g., all or a predetermined number of the highest confidencescores), and/or the corresponding treatment attributes. The alert may bedisplayed on the respective device and a user may view the treatmentattributes and corresponding confidence scores to make a treatmentdecision.

The data processing system may use the same calibration value tocalibrate the predictions for the neural network for any number ofpatients without any further training or adjustment of the calibrationvalue. For example, the data processing system may execute the neuralnetwork using a set of characteristics of a second patient receivingtreatment at the set of one or more second radiotherapy machines. Theneural network may output a confidence score associated with a treatmentattribute (e.g., the first treatment attribute or a different treatmentattribute that is associated with a highest confidence score). Theneural network may adjust the confidence score for the new treatmentattribute according to the same calibration parameter that was used tocalibrate the confidence score for the first patient. The dataprocessing system may compare the calibrated confidence score to othercalibrated predicted confidence scores and/or to a set of criteria(e.g., a threshold). Responsive to determining the criteria is satisfiedand/or that the confidence score is the highest of the predictedconfidence scores, the data processing system may output the calibratedconfidence score and/or the second treatment attribute to a userinterface indicating the second treatment attribute as the predictedtreatment attribute.

In addition to or instead of displaying the calibrated confidence scoresfor the potential treatment attributes to treat a patient, the dataprocessing system may transmit instructions to a radiotherapy machine ofthe second set of radiotherapy machines to adjust the machine'sconfiguration. For example, the data processing system may automaticallytransmit instructions to cause the radiotherapy machine to treat apatient for which the neural network predicted a field geometry setting(e.g., a field geometry setting associated with a highest confidencescore and/or a confidence score that exceeds a threshold). Theinstructions may include a flag or setting that causes the radiotherapymachine to treat the patient using the predicted field geometry setting.Upon receipt of the instructions and, in some cases, receipt of anindication (e.g., an input on a user interface) indicating the patientis positioned to be treated, the radiotherapy machine may automaticallytreat the patient using the predicted field geometry setting.Advantageously, because the confidence scores for the treatmentattributes are calibrated before they are compared to the threshold, thesystem can ensure that any automatic adjustments to the field geometrysettings of the radiotherapy machine, or any other radiotherapy machineattribute, may be accurate and not based on an inaccurate oroverconfident prediction.

Additionally or alternatively, instead of automatically treating thepatient with the predicted treatment attribute, the radiotherapy machinemay treat the patient with the predicted treatment attribute responsiveto receiving an input at a user interface (e.g., a user interfacedisplayed at clinic computer or an end-user computer). For example, thedata processing system may transmit an indication of the predictedtreatment attribute (e.g., predicted field geometry setting) and theassociated confidence score of the treatment attribute to theradiotherapy machine or a clinic computer or end-user computer. Theconfidence score and the treatment attribute may be displayed on adisplay of the radiotherapy machine or the clinic computer or end-usercomputer. A user (e.g., the patient, a clinician, a doctor, etc.) mayview and select the confidence score and/or treatment attribute.Responsive to receiving the selection, the radiotherapy machine mayadjust its configuration to treat the patient according to the selectedtreatment attribute.

For example, the data processing system may determine confidence scoresfor field geometry settings for different VMAT arcs. For instance, for aparticular patient's treatment, the data processing system may use thesystems and methods described herein to determine calibrated confidencescores for four unique VMAT arcs (e.g., accelerator arcs that can beused for treatment). For each arc, the data processing system mayfurther determine (e.g., with one or more other calibrated oruncalibrated machine learning models or the same machine learning modelthat determined the confidence scores for the respective arc) anddisplay confidence scores for characteristics of the arcs such ascollimator angle, couch angle, gantry endpoint, gantry starting point,isocenter location attributes (in each axis), VMAT type, X1 and X2 jawvalues, etc. An end-user (e.g., technician and/or treating oncologist orany other medical professional viewing the graphical user interface) mayadd, revise, and/or overwrite any of the depicted values and provide aninput that causes the respective radiotherapy machine to treat thepatient using the selected VMAT arc and the arc characteristics.

Additionally or alternatively, the data processing may transmit multiplecalibrated confidence scores and treatment attributes (e.g., apredetermined number of the highest confidence scores of the predictedscores and/or scores that exceed a threshold) to the radiotherapymachine, the clinic computer, or the end-user computer. In suchinstances, the confidence scores and associated treatment attributes mayeach be displayed so a user may select the attribute to use to configurethe radiotherapy machine or to otherwise treat the patient.

When the user performs an activity on the electronic platform, the dataprocessing system may track and record details of a user's activity. Forinstance, when a predicted result is displayed on a user's electronicdevice, the data processing system may monitor the user's electronicdevice to identify whether the user has interacted with the predictedresults by editing, deleting, accepting, or revising the results. Thedata processing system may also identify a timestamp of eachinteraction, such that the data processing system records the frequencyof modification, duration of revision/correction.

The data processing system may utilize an application programminginterface (API) to monitor the user's activities. The data processingsystem may use an executable file to monitor the user's electronicdevice. The data processing system may also monitor the electronicplatform displayed on an electronic device via a browser extensionexecuting on the electronic device. The data processing system maymonitor multiple electronic devices and various applications executingon the electronic devices. The data processing system may communicatewith various electronic devices and monitor the communications betweenthe electronic devices and the various servers executing applications onthe electronic devices.

The neural network may be trained using a supervised method usingpatient data of patients that were treated at one or more of the secondset of radiotherapy machines. For instance, during operation, the neuralnetwork may predict confidence scores for field geometry settings. Thedata processing system may calibrate the confidence scores, identify thehighest confidence score or that otherwise satisfies a threshold, anddisplay the calibrated confidence score and/or the corresponding fieldgeometry setting on a user interface. Users may either select thepredicted field geometry settings via the user interface to configurethe respective radiotherapy machine according to the predicted fieldgeometry setting or provide an input to indicate the predicted fieldgeometry setting was not used to treat the respective patient. Thedevice that received the user input may transmit a signal back to thedata processing system indicating the user input along with thepredicted field geometry setting and/or a patient identifier of thepatient that was treated. The data processing system may receive thesignal, identify the data that was used to make the prediction (e.g.,via the patient identifier), and label the data according to the userinput (e.g., a 1 to indicate the prediction was used to adjust theconfiguration of the radiotherapy machine and a 0 to indicate theprediction was not used). The data processing system may feed thelabeled data into the neural network for training. Similar methods ofsupervised may be used to train models that predict any type oftreatment attribute based on an input that indicates whether thepredicted treatment attribute was implemented.

Responsive to training the neural network with data sets of one or morepatients, the data processing system may recalibrate the neural networkby determining one or more new calibration parameters for the newlytrained neural network. The data processing system may determine newcalibration parameters because the previous calibration parameters weredetermined based on how the neural network was previously weighted andmay not be accurate for the new weights or parameters that result fromthe further training. The data processing system may determine thecalibration parameters for the neural network in the same manner asdescribed above. When calibrating the neural network, the dataprocessing system may use the same calibration data set or a differentcalibration data set that is similarly representative of the populationof patients that are treated at the set of one or more secondradiotherapy machines. The data processing system may determine the newcalibration parameters and make treatment attribute predictions forfuture patient treatments using the new calibration parameters until theneural network is further trained and/or recalibrated.

Additionally or alternatively, the data processing system may utilizemultiple neural networks or machine learning models to obtain aprediction for a treatment attribute for a patient. Each of the neuralnetworks may have been trained by patient data of patients that weretreated outside of the second set of radiotherapy machines and may beaccessed by the data processing system as described above. The dataprocessing system may use the same or different calibration data setsfor each neural network to determine calibration values for each neuralnetwork. For example, the data processing system may use the samecalibration data set to calibrate multiple neural networks that areassociated with treating patients at a radiotherapy clinic. However,because the neural networks may have been initially trained based on adifferent sets of training data, the calibrations parameter(s) for eachof the neural networks may be different. Upon being calibrated, the dataprocessing system may input the same patient characteristics of apatient to each of the neural networks to obtain confidence scores forone or more treatment attributes to use to treat the patient.

In one example, the data processing system may compare the outputconfidence scores from each of the neural networks with each other. Thedata processing system may identify the highest confidence score andselect the treatment attribute that is associated with the highestconfidence score for display and/or to automatically use to treat thepatient with the radiotherapy machine. Advantageously, by using multipleneural networks, the data processing system may identify the treatmentattribute for which a neural network is the most confident as thecorrect treatment attribute prediction and present a recommendation touse the treatment attribute for treatment. Using multiple neuralnetworks may provide the data processing system with more data to selecttreatment attributes, enabling more informed decisions. Further, becauseeach of the neural networks is individually calibrated, the highestcalibrated confidence score may be more likely to be accurate and/or beassociated with the correct treatment attribute than predictions thatare made based on multiple uncalibrated models or a calibrated model.

In another example, the data processing system may aggregate thecorresponding output confidence scores (e.g., confidence scores foridentical treatment attributes) of each of the neural networks todetermine the treatment attribute to predict. The data processing systemmay aggregate the confidence score outputs across neural networks andcompare the aggregated confidence scores of each treatment attributewith each other. The data processing system may select the treatmentattribute that is associated with the highest confidence score as thepredicted treatment attribute for display or to adjust the radiotherapymachine for treatment. Thus, the data processing system may crowdsourceconfidence score predictions from multiple calibrated neural networks toidentify treatment attribute predictions, further improving the accuracyof such predictions and avoiding the use of a single inaccurate and/oroverconfident confidence score prediction.

Using the systems and methods described herein, the data processingsystem can have a formalized approach to generate a treatment attributein a single automated framework based on various variables, parameters,and settings that depend on the patient, the patient's treatment, and/orthe clinic. The systems and methods described herein enable a centralserver or a processor associated with (e.g., located in) a local clinicto generate treatment attributes that are optimized for individualpatients based on the standard treatments at the clinic, replacing theneed to depend on a technician or doctor's subjective skills andunderstanding. The systems and methods may enable the central server orprocessor to do so without using training data from patients beingtreated at the clinic, which may not always be available.

Referring now to FIG. 3, a non-limiting example of a calibration dataset 300 is illustrated. The calibration data set 300 may include aplurality of calibration patient data sets 302 a-n (hereinafterdescribed as calibration patient data sets 302 or calibration patientdata set 302). The calibration patient data sets 302 may be fed into amachine learning model (e.g., an externally trained machine learningmodel) such as a neural network, as described above, to generate fieldgeometry predictions, or other predictions relating to RTTP treatment.The calibration data set 300 may include any number of calibrationpatient data sets 302. The calibration data set 300 may be consideredone or multiple data sets.

Each calibration patient data set 302 may include one or more patientcharacteristics or attributes indicating characteristics of a particularpatient. The characteristics or attributes may include valuesidentifying characteristics of the patient, how the patient is currentlybeing treated (e.g., radiation dose distribution and/or other aspects ofRTTP), TNM staging information, and/or the location on which the patientis experiencing a problem that may be treated through radiotherapy(e.g., the location, size, and/or shape of a tumor). The characteristicsor attributes may additionally or instead include image data of targetsor organs of the patient. In a non-limiting example, a patient data setmay include data relating to the gender, weight, height, dosedistribution, target organ, body mass index, geometry, etc., of apatient. The calibration patient data sets 302 may include anyattributes or characteristics of patients.

The calibration data set 300 may be representative of the patientpopulation that is treated at a particular radiotherapy clinic or byradiotherapy machines that are located within a geographical region. Forexample, different radiotherapy machines and/or radiotherapy clinics maytreat people with different types of problems and/or use differentmethods to treat such problems. Examples may include different machinesor clinics may treat different gender ratios, people with differenttypes of problems, different weight distributions, people that aresusceptible to different types and extremes of side effects, etc. Suchrepresentative data sets may be manually selected or selected by acomputer by identifying an average baseline for differentcharacteristics of patients and identifying patients that havecharacteristics that, upon being aggregated together, match or arewithin a threshold of the average baseline. Advantageously, because thecalibration data set 300 may be representative of the population that istreated at a clinic or a particular set of machines, the data set may beused to accurately calibrate a machine learning model to predicttreatment attributes to treat patients at the clinic or set ofradiotherapy machines. Systems that do not use such a representativecalibration data set may cause the machine learning model to beinaccurately calibrated, reducing the accuracy of confidence scorepredictions by the machine learning model.

Referring now to FIG. 4, a non-limiting example of sequences 400 forcalibrating a machine learning model and using the machine learningmodel to predict field geometry predictions to treat a patient isillustrated. The sequences 400 may include a sequence 402 for generatinga calibration parameter for a machine learning model and a sequence 404for using the calibration parameter to make field geometry setting (orother treatment attribute) predictions for a patient.

In the sequence 402, the calibration characteristics 406 may be inputinto a machine learning model 408. The calibration characteristics 406may be representative of a population of patients that are treated by aset of radiotherapy machines at one or more radiotherapy clinics. Forinstance, in a non-limiting example, the calibration characteristics 406may represent people that are treated at radiotherapy machines in thenorthwest region of the United States or at a local clinic within thesame region. Calibration characteristics 406 for individual patients maybe sequentially input or inserted into the machine learning model 408 bya computer, such as a computer at a clinic, or in the cloud. The machinelearning model 408 may output an RTTP prediction such as treatmentattribute predictions 410 indicating predicted treatment for each of thepatients.

The machine learning model 408 may be any machine learning model such asa neural network, random forest, support vector machine, etc. Themachine learning model 408 may have been trained to predict differenttreatment attributes such as treatment modalities, field geometrysettings, symptom likelihoods, etc. The machine learning model 408 mayhave been trained using data from patients that were treated atradiotherapy machines outside of the set of radiotherapy machinesdescribed above. A computer at a radiotherapy clinic (e.g., that isassociated with the set of radiotherapy machines) may access the machinelearning model 408 by retrieving or receiving the machine learning model408 (e.g., via an HTTP GET request) or by accessing a cloud database orserver in which the machine learning model 408 is stored. Upon accessingthe machine learning model 408, the computer may input the calibrationcharacteristics 406 into the machine learning model 408 to obtaintreatment attribute predictions 410.

Upon generating the treatment attribute predictions 410 for each of thepatients of the calibration characteristics 406, the computer may inputthe treatment attribute predictions 410 into a calibration model 412along with treatment attribute labels 414 representing the ground truthassociated with the treatment attribute predictions 410 (e.g., thecorrect labels that correspond to the calibration characteristics). Forinstance, the treatment attribute predictions 410 may be a vectorincluding binary values indicating whether patients should be treatedwith standard field geometry settings. The treatment attribute labels414 may be a vector of the same size indicating the correct prediction(e.g., the correct binary value) for each patient. The calibration model412 may be a machine learning model, such as a neural network, or anoptimization algorithm that is configured to receive the treatmentattribute predictions 410 and treatment attribute labels 414 and outputone or more calibration parameters 416 based on the treatment attributepredictions 410 and treatment attribute labels 414.

The calibration parameter 416 may be one or more calibration parametersthat may be used to calibrate confidence scores for different labels orclassifications that are output by the machine learning model 408. Forinstance, the calibration parameter 416 may be a temperature value thatmay be used in temperature scaling to adjust confidence scores that areoutput by the machine learning model 408. The calibration parameter 416may be any calibration value output by the calibration model 412. Thecalibration parameter 416 may be used to output any predictions outputby the machine learning model 408, thus calibrating predictions made bythe machine learning model 408.

After obtaining the calibration parameter 416, at sequence 404, patientcharacteristics 418 of a patient being treated by one of the same set ofradiotherapy machines for which the machine learning model 408 wascalibrated may be input into the machine learning model 408. The patientcharacteristics 418 may include values of the same attributes orcharacteristics as the patient characteristics of the calibrationcharacteristics 406. The machine learning model 408 may receive thepatient characteristics 418 and output a treatment attribute prediction420 including one or more confidence scores for different treatmentattributes (e.g., field geometry settings). The computer may input thetreatment attribute prediction 420 into a prediction adjuster 422 withthe previously determined calibration parameter 416. The predictionadjuster 422 may comprise instructions executable by one or moreprocessors that causes the processors to perform one or more operationson the treatment attribute prediction 420 using the calibrationparameter 416 to output a calibrated treatment attribute prediction 424including calibrated confidence scores of treatment attribute prediction420 output by the machine learning model 408.

The computer may obtain the calibrated treatment attribute prediction424 and perform several actions using the prediction. For example, thecomputer may be connected to a radiotherapy machine of the set ofradiotherapy machines and the treatment attribute prediction 424 mayinclude confidence scores for different field geometry settings. Thecomputer may compare the confidence scores of the calibrated treatmentattribute prediction 424 to a threshold (e.g., a predeterminedthreshold). Responsive to identifying a confidence score that exceedsthe threshold, the computer may identify the field geometry settingassociated with the confidence score that exceeds the threshold andadjust or operate the radiotherapy machine to treat the patient usingthe field geometry setting. In another example, the computer may displaythe one or more confidence scores and associated field geometry settingsto a user such as the patient, a doctor, or a clinician. The user mayeither agree with the prediction and provide an input that causes theradiotherapy machine to adjust its configuration according to thedisplayed setting or provide an input rejecting the displayed setting.Such may be advantageous when the machine learning model 408 is notconfident above a threshold about a specific treatment (e.g., the systemmay display the setting responsive to determining the calibratedconfidence score does not exceed a threshold) or if a patient is morecomfortable making decisions themselves rather than relying on acomputer.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedherein may be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and steps have been described above generally in terms oftheir functionality. Whether such functionality is implemented ashardware or software depends upon the particular application and designconstraints imposed on the overall system. Skilled artisans mayimplement the described functionality in varying ways for eachparticular application, but such implementation decisions should not beinterpreted as causing a departure from the scope of this disclosure orthe claims.

Embodiments implemented in computer software may be implemented insoftware, firmware, middleware, microcode, hardware descriptionlanguages, or any combination thereof. A code segment ormachine-executable instructions may represent a procedure, a function, asubprogram, a program, a routine, a subroutine, a module, a softwarepackage, a class, or any combination of instructions, data structures,or program statements. A code segment may be coupled to another codesegment or a hardware circuit by passing and/or receiving information,data, arguments, parameters, or memory contents. Information, arguments,parameters, data, etc. may be passed, forwarded, or transmitted via anysuitable means including memory sharing, message passing, token passing,network transmission, etc.

The actual software code or specialized control hardware used toimplement these systems and methods is not limiting of the claimedfeatures or this disclosure. Thus, the operation and behavior of thesystems and methods were described without reference to the specificsoftware code being understood that software and control hardware can bedesigned to implement the systems and methods based on the descriptionherein.

When implemented in software, the functions may be stored as one or moreinstructions or code on a non-transitory computer-readable orprocessor-readable storage medium. The steps of a method or algorithmdisclosed herein may be embodied in a processor-executable softwaremodule, which may reside on a computer-readable or processor-readablestorage medium. A non-transitory computer-readable or processor-readablemedia includes both computer storage media and tangible storage mediathat facilitate transfer of a computer program from one place toanother. A non-transitory processor-readable storage media may be anyavailable media that may be accessed by a computer. By way of example,and not limitation, such non-transitory processor-readable media maycomprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage,magnetic disk storage or other magnetic storage devices, or any othertangible storage medium that may be used to store desired program codein the form of instructions or data structures and that may be accessedby a computer or processor. Disk and disc, as used herein, includecompact disc (CD), laser disc, optical disc, digital versatile disc(DVD), floppy disk, and Blu-ray disc where disks usually reproduce datamagnetically, while discs reproduce data optically with lasers.Combinations of the above should also be included within the scope ofcomputer-readable media. Additionally, the operations of a method oralgorithm may reside as one or any combination or set of codes and/orinstructions on a non-transitory processor-readable medium and/orcomputer-readable medium, which may be incorporated into a computerprogram product.

The preceding description of the disclosed embodiments is provided toenable any person skilled in the art to make or use the embodimentsdescribed herein and variations thereof. Various modifications to theseembodiments will be readily apparent to those skilled in the art, andthe principles defined herein may be applied to other embodimentswithout departing from the spirit or scope of the subject matterdisclosed herein. Thus, the present disclosure is not intended to belimited to the embodiments shown herein but is to be accorded the widestscope consistent with the following claims and the principles and novelfeatures disclosed herein.

While various aspects and embodiments have been disclosed, other aspectsand embodiments are contemplated. The various aspects and embodimentsdisclosed are for purposes of illustration and are not intended to belimiting, with the true scope and spirit being indicated by thefollowing claims.

What we claim is:
 1. A method comprising: accessing, by one or moreprocessors, a neural network trained based on a first set of datagenerated from characteristic values of a first set of patients thatreceived treatment at a set of one or more first radiotherapy machines;executing, by the one or more processors, the neural network using asecond set of data comprising characteristic values of a second set ofpatients receiving treatment at a set of one or more second radiotherapymachines to output a set of treatment attribute predictions, the secondset of data having corresponding labels indicating expected treatmentattribute predictions; executing, by the one or more processors, acalibration model using the set of treatment attribute predictions andlabels indicating expected treatment attribute predictions to output acalibration value; executing, by the one or more processors, the neuralnetwork using a set of characteristics of a first patient of the secondset of patients receiving treatment at the set of one or more secondradiotherapy machines to output a first confidence score associated witha first treatment attribute; and adjusting, by the one or moreprocessors, the first confidence score according to the calibrationvalue to predict the first treatment attribute.
 2. The method of claim1, further comprising: executing, by the one or more processors, theneural network using a set of characteristics of a second patient of thesecond set of patients receiving treatment at the set of one or moresecond radiotherapy machines to output a second confidence scoreassociated with a second treatment attribute and adjusting the secondconfidence score according to the calibration value to predict thesecond treatment attribute.
 3. The method of claim 1, wherein predictingthe first treatment attribute comprises comparing the adjusted firstconfidence score to a threshold and predicting the first treatmentattribute responsive to determining the adjusted first confidence scoreexceeds the threshold.
 4. The method of claim 1, further comprising:adjusting, by the one or more processors, a configuration of a secondradiotherapy machine of the set of one or more second radiotherapymachines according to the first treatment attribute.
 5. The method ofclaim 1, further comprising: rendering, by the one or more processors,the adjusted first confidence score and the first treatment attribute ona display; receiving, by the one or more processors, a user inputselecting the first treatment attribute; and responsive to receiving theuser input, adjusting, by the one or more processors, a configuration ofa second radiotherapy machine of the set of second one or more secondradiotherapy machines based on the first treatment attribute.
 6. Themethod of claim 1, wherein the neural network is a first neural networkand the calibration value is a first calibration value, furthercomprising: executing, by the one or more processors, a second neuralnetwork using the set of characteristics of the first patient to outputa second confidence score for a second treatment attribute; andadjusting, by the one or more processors, the second confidence scoreassociated with the second treatment attribute according to a secondcalibration value corresponding to the second neural network, the secondcalibration value generated based on the second set of data; whereinpredicting the first treatment attribute is performed responsive to thefirst confidence score exceeding the second confidence score.
 7. Themethod of claim 1, wherein the first treatment attribute is an angleassociated with a couch or a gantry of a radiotherapy machine.
 8. Themethod of claim 1, wherein predicting the first treatment attributecomprises: executing, by the one or more processors, a plurality ofneural networks using the set of characteristics of the first patient tooutput confidence scores for a plurality of treatment attributes, eachneural network associated with a different calibration value; adjusting,by the one or more processors, the output confidence scores of each ofthe plurality of neural networks according to the calibration valueassociated with the respective neural network; and predicting, by theone or more processors, the first treatment attribute by aggregating theadjusted confidence scores that correspond to individual treatmentattributes.
 9. The method of claim 1, further comprising: adjusting, bythe one or more processors, the calibration value responsive to trainingthe neural network according to a supervised learning algorithm from atraining set of values labeled according to whether a radiotherapymachine adjusted its configuration based on a predicted treatmentattribute.
 10. The method of claim 1, wherein the calibration model isone of a machine learning model or a mathematical optimizationalgorithm.
 11. The method of claim 1, further comprises: predicting, bythe one or more processors, a third confidence score associated with athird treatment attribute by executing the neural network using a set ofcharacteristics of a third patient receiving treatment at the secondradiotherapy clinic; adjusting, by the one or more processors, the thirdconfidence score according to the calibration value; and responsive todetermining the adjusted third confidence score does not exceed athreshold, rendering, by the one or more processors, the thirdconfidence score and an indication that the third confidence score doesnot exceed the threshold on a display.
 12. The method of claim 1,wherein the neural network is trained based only on data associated withthe first set of patients.
 13. A system comprising: one or moreprocessors in communication with a radiotherapy machine, the processorconfigured to execute instructions to: access a neural network trainedbased on a first set of data generated from characteristic values of afirst set of patients that received treatment at a set of one or morefirst radiotherapy machines; execute the neural network using a secondset of data comprising characteristic values of a second set of patientsreceiving treatment at a set of one or more second radiotherapy machinesto output a set of treatment attribute predictions, the second set ofdata having corresponding labels indicating expected treatment attributepredictions; execute a calibration model using the set of treatmentattribute predictions and labels indicating expected treatment attributepredictions to output a calibration value; execute the neural networkusing a set of characteristics of a first patient of the second set ofpatients receiving treatment at the set of one or more secondradiotherapy machines to output a first confidence score associated witha first treatment attribute; and adjust the first confidence scoreaccording to the calibration value to predict the first treatmentattribute.
 14. The system of claim 13, wherein the processor is furtherconfigured to: execute the neural network using a set of characteristicsof a second patient of the second set of patients receiving treatment atthe set of one or more second radiotherapy machines to output a secondconfidence score associated with a second treatment attribute andadjusting the second confidence score according to the calibration valueto predict the second treatment attribute.
 15. The system of claim 13,wherein the processor is configured to predict the first treatmentattribute by comparing the adjusted first confidence score to athreshold and predicting the first treatment attribute responsive todetermining the adjusted first confidence score exceeds the threshold.16. The system of claim 13, wherein the processor is further configuredto: adjust configuration of a second radiotherapy machine of the set ofone or more second radiotherapy machines according to the firsttreatment attribute.
 17. The system of claim 13, wherein the processoris further configured to: render the adjusted first confidence score andthe first treatment attribute on a display; receive a user inputselecting the first treatment attribute; and responsive to receiving theuser input, adjust a configuration of a second radiotherapy machine ofthe set of second one or more second radiotherapy machines based on thefirst treatment attribute.
 18. The system of claim 13, wherein theneural network is a first neural network and the calibration value is afirst calibration value, and wherein the processor is further configuredto: execute a second neural network using the set of characteristics ofthe first patient to output a second confidence score for a secondtreatment attribute; and adjust the second confidence score associatedwith the second treatment attribute according to a second calibrationvalue corresponding to the second neural network, wherein the processoris configured to predict the first treatment attribute responsive to thefirst confidence score exceeding the second confidence score.
 19. Thesystem of claim 13, wherein the first treatment attribute is an angleassociated with a couch or a gantry of a radiotherapy machine.
 20. Thesystem of claim 13, wherein the processor is configured to predict thefirst treatment attribute by: executing a plurality of neural networksusing the set of characteristics of the first patient to outputconfidence scores for a plurality of treatment attributes, each neuralnetwork associated with a different calibration value; adjusting theoutput confidence scores of each of the plurality of neural networksaccording to the calibration value associated with the respective neuralnetwork; and predicting the first treatment attribute by aggregating theadjusted confidence scores that correspond to individual treatmentattributes.